Image processing apparatus and image processing method

ABSTRACT

An image processing apparatus, comprising a memory that stores first image data, and a processor that includes an image associated information processing section, wherein the image associated information processing section, for the image data of a single frame that has been taken at a plurality of shooting conditions, within the first image data that has been stored in the memory, acquires image region information, relating to an image region in which shooting is carried out at different shooting conditions, and image associated information of the image region, associates the image region information and the image associated information and subjects the first image data to image processing, and generates second image data.

CROSS-REFERENCE TO RELATED APPLICATIONS

Benefit is claimed, under 35 U.S.C. § 119, to the filing date of priorJapanese Patent Application No. 2016-116299 filed on Jun. 10, 2016. Thisapplication is expressly incorporated herein by reference. The scope ofthe present invention is not limited to any requirements of the specificembodiments described in the application.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus and imageprocessing method for carrying out processing to associate image data,and image associated information relating to that image data.

2. Description of the Related Art

In an imaging apparatus such as a digital camera there is one set ofshooting conditions (for example, shutter speed, aperture value, ISOsensitivity etc.) when shooting a single frame. However, in recentyears, it has been proposed to carry out shooting with a differentshutter speed for every pixel or region, by changing exposure time foreach pixel or line of an image sensor, as shown in Japanese patentlaid-open No. 2011-044966 (hereafter referred to as “patent publication1”). Also, Japanese patent laid-open No. 2011-130167 proposes an imagingapparatus that applies different image processing gain for each pixel oreach image region of an image for a single taken frame, and generatesimages that have been taken with effectively different sensitivity (ISOsensitivity). In this way it is proposed to take a single frame at aplurality of shooting conditions.

Also, a plurality of images are combined and creation of an image of asingle frame is carried out. For instance, an imaging apparatus isavailable on the market that acquires a plurality of images while makingshooting conditions such as shutter speed and focus position different,and by combining the plurality of images, functions such as HDRcombination where dynamic range is extended, and focus stacking wheredepth of focus is increased, are provided. By forming the image data fora single frame under various shooting conditions in this way, andcombining, it is possible to obtain an image with an effect applied thatcan not be obtained with normal shooting.

Also, with digital cameras that are available on the market, and imageprocessing apparatus etc. for processing digital camera images,information relating to images, such as shooting conditions, is madeassociated information (metadata) of the image data, and it is possibleto apply processing and to store this metadata. Further, with a digitalcamera, when performing playback display of alive view image or an imagethat has already been taken, the images may be displayed with theseitems of associated information overlaid.

SUMMARY OF THE INVENTION

An image processing apparatus of a first aspect of the present inventioncomprises a memory that stores first image data, and a processor thatincludes an image associated information processing section, wherein theimage associated information processing section, for the image data of asingle frame that has been taken at a plurality of shooting conditions,within the first image data that has been stored in the memory, acquiresimage region information, relating to an image region in which shootingis carried out at different shooting conditions, and image associatedinformation of the image region, associates the image region informationand the image associated information, and subjects the first image datato image processing, and generates second image data.

An image processing method of a second aspect of the present inventioncomprises carrying out shooting with different shooting conditions andstores image data of a single frame, extracts image associatedinformation for image data of the single frame, and carries outassociation of the image data for the single frame with the imageassociated information for every image region.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram mainly showing the electrical structure of acamera of a first embodiment of the present invention.

FIG. 2 is a flowchart showing operation of the camera of a firstembodiment of the present invention.

FIG. 3A and FIG. 3B are drawings for describing a front curtainelectronic shutter of the camera of the first embodiment of the presentinvention, and gain application to achieve an effect that is similar tothis.

FIG. 4A and FIG. 4B are drawings showing one example of displaying aplurality of shooting conditions, in the camera of the first embodimentof the present invention.

FIG. 5A and FIG. 5B are drawings showing one example of image outputinformation processing when shooting at a plurality of shootingconditions, in the camera of the first embodiment of the presentinvention.

FIG. 6A and FIG. 6B are drawings showing another example of image outputinformation processing when shooting at a plurality of shootingconditions, in the camera of the first embodiment of the presentinvention.

FIG. 7A and FIG. 7B are drawings for explaining storage of associatedinformation in a case where shooting has been performed at a pluralityof shooting conditions in the camera of the first embodiment of thepresent invention.

FIG. 8 is a flowchart showing operation of the camera of a secondembodiment of the present invention.

FIG. 9 is a drawing showing one example of displaying a plurality ofshooting conditions, in the camera of the second embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

An example where a digital camera is adopted as a preferred embodimentof the present invention will be described in the following. This camerahas an imaging section, with a subject image being converted to imagedata by this imaging section, and the subject image being subjected tolive view display on a display section arranged on the rear surface ofthe camera body based on this converted image data. A photographerdetermines composition and photo opportunity by looking at a live viewdisplay. At the time of a release operation image data is stored in astorage medium. Image data that has been stored in the storage mediumcan be subjected to playback display on the display section if playbackmode is selected.

An image sensor of this camera also has an electronic shutter, and canmake shutter speed (in other words, exposure time) different for everygiven region on an image sensor surface (refer to FIG. 3A and FIG. 3B,that will be described later). Specifically, when carrying out shootingof a single frame it is possible to carry out shooting at a plurality ofshooting conditions. With this embodiment, when carrying out live viewdisplay, it is possible to display shooting conditions for each ofrespective regions (refer to FIG. 4A and FIG. 4B that will be describedlater). Also, when storing shooting conditions in image data, it ispossible to store shooting conditions for each region (referred to FIG.7B that will be described later).

FIG. 1 is a block diagram mainly showing the electrical structure of acamera of a first embodiment of the present invention. The camera ofthis embodiment comprises an imaging section 10, an image processingsection 20, a system controller 21 and a bus 30, with each section beingconnected to this bus. It should be noted that with this embodiment, alens 12 and an aperture 13 included in the imaging section 10 areintegrally formed with the camera body, but this integral structure isnot limiting and it is also possible to provide the lens and aperturewithin an interchangeable lens for use with an interchangeable lenscamera.

Inside the imaging section 10 there are provided a lens 12, aperture 13,shutter 14 and image sensor 15. The lens 12 is an optical lens forforming an optical image, and has a focus lens and a zoom lens. Theaperture 13 is provided on the optical axis of the lens 12. Thisaperture 13 has opening diameter varied by a drive control section 11,and amount of light that passes through the aperture 13 is controlled.

The shutter 14 is also provided on the optical axis of the lens 12. Thisshutter 14 is a mechanical shutter, and the shutter 14 controls time forwhich light flux passes through the lens 12, namely the time for whichlight is incident on the image sensor 15 (exposure time, shutter speed).It should be noted that when carrying out live view display the shutter14 is in an open state. Also, the aperture 13 is normally set to awide-open F value.

The image sensor 15 is provided on the optical axis of the lens 12, andclose to a position where an optical image of a subject is formed. Theimage sensor 15 includes an image sensor such as a CMOS image sensor ora CCD image sensor, and converts an optical image of a subject that hasbeen formed by the lens 2 into a pixel signal for every pixel, beforeoutputting this pixel signal to the bus 30. The bus 30 is a signal linefor exchanging signals between each block. The image sensor 15 functionsas an image sensor. This image sensor is capable of varying shutterspeed for every pixel or every line (refer to FIG. 3A and FIG. 3B, thatwill be described later).

The lens 12, aperture 13, shutter 14 and image sensor 15 are connectedto a drive control section 11. The drive control section 11 carries outdrive control for the focus lens of the lens 12, carries out aperturevalue control for the aperture 13, and carries out opening and closingcontrol of the shutter 14 (shutter speed control) in accordance withdrive signals from the system controller 21. Also, the drive controlsection 11 has an imaging control circuit for the image sensor 15, andcarries out charge accumulation control for the image sensor 15 andpixel signal readout control etc.

The image processing section 20 has a processor for image processing andis connected to the bus 30, and is provided with an image associatedinformation processing section 200, gain application section 26, imagecombination section 27, development processing section 28, and displayimage generating section 29. The image associated information processingsection 200 is provided with a region information processing section201, shooting condition processing section 202, image output informationprocessing section 203 and image data generating section 204.

The image associated information processing section 200 is part of theabove-described processor, and functions as an image associatedinformation processing section that performs processing to associateimage region information relating to image regions where shooting iscarried out at different shooting conditions (for example, image regionsdetermined by a boundary line LINE in FIG. 4B and FIG. 5A, and imageregions determined by blocks shown in FIG. 6A and FIG. 7A) and imageassociated information of the image regions, for image data of one framethat has been taken at a plurality of shooting conditions.

Image data for one frame that has been taken at a plurality of shootingconditions is image data that has been formed by the image sensor.Shutter speed is treated as image associated information (refer to FIG.3A and FIG. 3B that will be described later). Image associatedinformation is at least one of shutter speed, ISO sensitivity, focallength and exposure control value (EV value), which are shootingconditions for each image region. Image associated information is atleast one of a statistical value for pixel output (histogram, EV value),exposure offset amount (EV value) on subject distance, which are valuesthat are calculated using pixel output contained in each image region.

The region information processing section 201 is a part of theabove-described processor, and can divide the imaging surface of theimage sensor 15 into a plurality of regions either automatically or inaccordance with a user command. For image data of one frame (including acombined image in a second embodiment that will be described later), ina case where shooting conditions and image output information aredifferent for every pixel address or region of an image, information onthese pixel addresses or image regions is extracted, and associated withshooting conditions and image output information.

The shooting condition processing section 202 is part of the abovedescribed processor, and for image data of one frame (including acombined image in the second embodiment that will be described later) ina case where shooting conditions are different for every pixel addressor every image region, extracts shooting conditions for each of thesepixel addresses or image regions. As shooting conditions the shootingcondition processing section 202 may extract shutter speed, sensitivity(gain), focal length (zoom information), aperture value etc., and mayalso extract shooting date, whether or not flash was used, exposurecontrol information (under, correct, over), art filter settings, or WB(white balance) settings as shooting conditions. These shootingconditions are calculated by the system controller 21 based on subjectbrightness for every region etc., and so the shooting conditionprocessing section 202 extracts shooting conditions that have beencalculated by the system controller 21. An example of shootingconditions that have been extracted by the shooting condition processingsection 202 will be described later using FIG. 4A and FIG. 4B.

The image output information processing section 203 is part of the abovedescribed processor, and for image data of one frame (including acombined image in the second embodiment that will be described later) ina case where image output information is different for every pixeladdress or every image region, extracts image output information foreach of these pixel addresses or image regions. As image outputinformation, it is possible to extract histogram information showingdistribution of subject brightness values, and also exposure (offsetfrom proper exposure, EV values), and if AF (Auto Focus) is set andimage plane phase difference AF is adopted it is possible to extractinformation on subject distance. An example of image output informationthat has been extracted by the image output information processingsection 203 will be described later using FIG. 5A and FIG. 5B.

The image data generating section 204 is part of the above describedprocessor, and associates region information with shooting condition andimage output information, and stores this data that has been associatedtogether with the image data in an external memory 23. A data structurefor image data and data that has been associated with this image datawill be described later using FIG. 7B. The image data generating section204 functions as an image data generating section that stores imageregion information that has been associated with image data by an imageinformation association processing section, and image associatedinformation for image regions, as metadata of the image data.

The gain application section 26 has a gain application circuit, andapplies a gain value to the image data parts has been output from theimage sensor 15. Pixel output is multiplied in accordance with the gainvalue. The gain application section 26 not only applies gain valuesuniformly to all pixels of image data of a single frame, but can alsoapply different gain values to each region that has been divided by theregion information processing section 201. The gain application section26 functions as a gain application section that supplies gain to imageoutputs of every pixel or every line, and treats gain values as imageassociated information.

It should be noted that application of a gain value to the image data isnot limited to being carried out by the gain application section 26 andmay be carried out in the image sensor 15. Specifically, a gainapplication circuit may be provided internally to the image sensor 15,and application of the gain value to a pixel signal carried out in theimage sensor 15.

The image combination section 27 has an image combining circuit andcombines image data of a plurality of frames. As this combinationprocessing, there are additive combination processing, averagingcombination processing, relatively bright combination processing,relatively dark combination processing, HDR combination processing(combination of images that have been taken while changing exposurevalue), focus stacking processing (combination of images that have beentaken while shifting focus position), spreading a plurality of imagedata over an image region and creating a combined photograph for oneframe (photo story) etc. The image combination section 27 functions asan image combination section that combines image data of a plurality offrames, and image data of one frame that has been taken at a pluralityof shooting conditions is image data that is combined by the imagecombination section.

The development processing section 28 has a development processingcircuit, and applies development processing to image data that has beenread out from the image sensor 15 or to image data that has beentemporarily held in internal memory 22. As development processing thereare demosaicing processing, noise removal processing, WB balanceprocessing, edge enhancement processing etc. The development processingsection 28 also carries out image data compression and expansionprocessing such as JPEG and MPEG etc. In the event that the number ofpixels of the image sensor 15 and the display section 24 is different,resizing processing is also carried out to increase or reduce the numberof pixels so as to match the number of pixels of the display section 24.

The display image generating section 29 has a display image generatingcircuit, and generates a live view image or image for playback displayof a taken image, and generates operation menu images etc. The generatedimage is displayed on the display section 24. Further, with thisembodiment an image that has shooting associated information that hasbeen extracted by the image output information processing section 203superimposed on a live view or playback image is generated, anddisplayed on the display section 24.

It should be noted that in this embodiment, the image processing section20 is a dedicated image processing processor comprising an arithmeticcircuit that carries out image processing calculations, a CPU (CentralProcessing Unit) and a memory that stores programs. Within the imageprocessing section 20, the gain application section 26, imagecombination section 27, development processing section 28 and displayimage generating section 29 are comprised of arithmetic circuits, whilethe remaining sections are executed by the CPU using programs. However,this structure is not limiting, and it is also possible to have astructure where, for example, image processing calculations are handledin a general purpose signal processing processor such as a digitalsignal processor (DSP), based on image processing programs. Certainsections may also be implemented by the system controller 21, that willwe described later, in accordance with programs.

The internal memory 22 has an electrically rewritable volatile memory,for example, a DRAM (Dynamic Random Access Memory), and an electricallyrewritable nonvolatile memory, for example, flash memory. The internalmemory 22 stores and holds image data and control programs for carryingout overall camera control etc. The internal memory 22 functions as astorage section for storing image data.

An external memory 23 has a portable memory medium, such as SD card orCF card, and this memory medium can be removed from and fitted into thecamera body. It is possible to carry out storage of image data to theexternal memory 23, and the previously described data that has beenassociated with the image data is also stored together with the imagedata. A user of this device can remove the external memory 23 from thecamera body, and carry out playback of image data in another device.

The display section 24 is a TFT (Thin Film Transistor) liquid crystaldisplay arranged on a rear surface part of the camera body, or an EVF(Electronic View Finder) that can be viewed via an eyepiece, anddisplays information relating to operations of this device, image data,and information relating to image data etc. The display section 24functions as a display that displays image region information and imageassociated information superimposed on image data.

An input section 25 has button type and switch type operation membersand a touch panel etc., with operating states of the operation membersbeing detected and detection results output to the system controller 21.The system controller 21 carries out control of the camera in accordancewith input states to this input section 25. Using a touch panel or thelike it is possible to carry out designation of regions, which will bedescribed later. It is also possible to carry out setting of shootingconditions (for example setting of shutter speed and ISO sensitivity)and to carry out image processing settings (WB setting, noise reductionprocessing, art filter), for every region. The input section 25functions as a shooting condition input section that receives inputs ofshooting conditions from the photographer, and sets shooting conditionsfor each of a plurality of image regions in accordance with signals fromthe shooting condition input section.

The system controller 21 has a CPU (Central Processing Unit) andperipheral circuits and memory for the CPU. The CPU implements theoverall control of the camera by controlling each of the sections withinthe camera 1 in accordance with programs stored in memory.

Next, operation of the camera of this embodiment will be described usingthe flowchart shown in FIG. 2. This flowchart is executed by the CPUwithin the system controller 21 controlling each of the sections withinthe camera in accordance with programs stored in the internal memory 22.

If the flowchart of FIG. 2 is commenced, first live view is displayed(S1). Here, the aperture 13 is set to a wide-open aperture value and theshutter 14 is placed in an open state, and a subject image is displayedon the display section 24 based on image data from the image sensor 15.Readout of the image sensor 15 carried out by the drive control section11 is carried out in normal mode.

Next, AE and AF (ranging) are carried out (S3). Here, in order to carryout AE (auto exposure: automatic exposure control), subject brightnessis detected based on image data from the image sensor 15. AF (autofocus) is also carried out. In the event that the image sensor 15 isutilizing an image plane phase difference imager, ranging is carried outover the entire pixel region. Subject brightness and ranging are carriedout in increments of each region designated in the next step and thedata is associated with its respective region.

Once AE and AF have been carried out, next region designation is carriedout (S5). Within a screen corresponding to the imaging surface of theimage sensor 15 is divided into a plurality of regions. Division is thedivision of an image into regions by the user with operation members ofthe input section 25, such as a touch panel. As well as manual settingby the user, this designation of regions may be carried outautomatically based on subject brightness and ranging results. In theexample shown in FIG. 4B, the user performs designation by moving adividing line LINE up and down. It is possible to set shootingconditions and image processing for each region that has been designated(refer to S7-S13).

Once region designation has been carried out, next setting of shutterspeed is carried out (S7). As was described using FIG. 3A and FIG. 3B,the image sensor 15 of this embodiment can change shutter speed for eachregion (each pixel). Shutter speed is therefore set for each region thatwas set in step S5. At the time of this setting shutter speed may bedetermined automatically so as to achieve optimal exposure based onsubject brightness for each region, or may be set manually by the user.

If shutter speed has been set, next setting of ISO sensitivity iscarried out (S9). Here, the gain application section 26 applies a gainvalue corresponding to ISO sensitivity for every region that was set instep S5. It should be noted that as was described previously, in a casewhere the image sensor 15 has a function to set gain for every region,setting of ISO sensitivity may also be carried out in the image sensor15.

Once setting of ISO sensitivity has been carried out, next setting of WB(white balance) is carried out (S11). Here the development processingsection 28 carries out setting of WB for each region that was set instep S5. In a case where light sources of various color temperatures areshown within the screen, it is possible to achieve a picture thatappears natural by setting WB for every region in accordance with thatlight source.

If setting of WB has been carried out, next setting of image processingis carried out (S13). Settings for noise reduction and settings for artfilters are changed for every region of the image that was set in stepS5, in accordance with the user's preference.

Once setting of image processing has been carried out, next shootingcondition processing is carried out (S15). Here, the shooting conditionprocessing section 202 extracts shooting conditions that have been setby the system controller 21 for every region that was set in step S5.

Once processing for shooting conditions has been carried out, next imageoutput information processing is carried out (S17). Here the imageoutput information processing section 203 extracts image outputinformation from a live view image, for every region that was set instep S5. The image output information, as was described previously, is ahistogram showing subject brightness distribution, EV value representingoffset from proper exposure value etc.

Once image output information processing has been carried out, nextregion information processing is carried out (S19). Here, in a casewhere shooting conditions (step S15) and image output information (stepS17) that were set for each region (each pixel) are different, theseshooting conditions and image output information associated with eachother.

Once region information processing has been carried out, next generationof a display image is carried out (S21). Here the display imagegenerating section 29 generates an image in which image regioninformation, and shooting conditions and image output information, issuperimposed on the live view image. It should be noted that the liveview image is created in accordance with shooting conditions etc. thathave been set. For example, in a case where shutter speed has beenshifted from the correct shutter speed towards a faster shutter speed, alive view image may be generated with the electronic shutter of theimage sensor 15 shifted towards a higher speed.

If generation of a display image has been carried out, live view displayis next carried out (S23). Here, an image that was generated by thedisplay image generating section 29 in step S21 is displayed as a liveview image. The user is able to grasp what shooting conditions were setfor each of the plurality of regions and what type of output resultedfor each region.

Once live view display has been carried out, it is next determinedwhether or not a release switch is on (S23). In a case where the userobserves a live view image that has been displayed on the displaysection 24 and has achieved a desired composition, and in a case where,while looking at shooting conditions and image output information forevery image region that have been superimposed on the live view image,shooting conditions etc. are adjusted to achieve a desired composition,a release button is pressed down fully. If a release switch is on as aresult of the release button being pressed down fully, shooting iscommenced. In a case where the release switch has not been turned onprocessing returns to step S3 and the previously described operationsare executed. In this case the input section 25 detects operating statesof the operation members that have been operated by the user andreflects detection results in actual operation.

If the result of determination in step S23 is that shooting is to becommenced, the system controller 21 controls aperture value of theaperture 13, controls shutter speed of the shutter 14, and controlsimaging by the image sensor 15. It should be noted that in a case wherethe shutter speed is different for every region, the electronic shutterof the image sensor 15 may be utilized in addition to the shutter 14.Specifically, the longest shutter speed is controlled using the shutter14, and for regions where the shutter speed shorter than this theshutter speed is controlled using the electronic shutter. For shootingconditions other than shutter speed also, imaging operations areexecuted in accordance with conditions that have been set.

If an exposure time determined by the shutter speed has elapsed, readout of image data is carried out (S25). Here the drive control section11 performs readout of image data from the image sensor 15, outputs tothe bus 30, and temporarily stores in the internal memory 22. It shouldbe noted that at the time of display of a live view image in step S1 orS23, the number of pixels read out is small, but with actual shooting ina case where it has been determined in step S23 that the release switchis on, the number of pixels is greater than at the time of live viewdisplay and high definition image data is acquired.

If read out of image data has been performed, next gain is applied tothe image data (S27). Here the gain application section 26 appliesgained to the image data. In a case where a different gain has been setfor each region (each pixel), gain is applied in accordance with thesettings. It should be noted that in a case where it is possible toapply gain when reading out image data from the image sensor 15, gainmay be applied when reading out from the image sensor 15 in step S25.

If gain has been applied, next development processing is carried out(S29). Here the development processing section 28 subjects image data toprocessing. In a case where different image processing settings (WB, artfilter etc.) have been set for each region (each pixel), imageprocessing is carried out in accordance with the settings.

Once development processing has been carried out, next shootingcondition processing is carried out (S31). Here the shooting conditionprocessing section 202 extracts shooting conditions for at the time ofshooting (shutter speed, gain etc.) from the system controller 21. Also,image processing settings that have been set in the image data are alsoextracted by the development processing section 28. At this time, in theevent that ranging is being carried out, focus position information maybe extracted as a shooting condition. If region designation is carriedout and shooting conditions are different for each region (each pixel),shooting conditions are extracted for each region.

Once shooting condition processing has been carried out, next imageoutput information processing is carried out (S33). Here, the imageoutput information processing section 203 calculates image outputinformation (histogram information, exposure (offset from properexposure, EV value), noise value) for every region (every pixel) thathas been set from image data. In the event that region designation hasnot been performed, image output information for the overall image maybe calculated.

Once image output information processing has been carried out, nextregion information processing is carried out (S35). Here, when regionshave been manually designated by the user, or regions have beenautomatically designated, if shooting has been performed with differentshooting conditions in each region (each pixel), the region informationprocessing section 201 associates shooting conditions and image outputinformation for each of the plurality of regions (pixels), and allocatesto header data (EXIF data etc.) as metadata of the image data.

Once region information processing has been carried out, next storage ofimage data is carried out (S37). Here, image data is stored in theexternal memory 23. Also, when storing image data, metadata of the imagedata that was allocated to the header data in step S35 is also stored.

If storage of image data has been carried out, generation of a displayimage is carried out (S39). Here the display image generating section 29generates an image in which shooting conditions or image outputinformation for every region (every pixel) has been superimposed onimage data that has been taken.

If a display image has been generated, next a playback image isdisplayed (S41). Here, an image that was generated in step S39 isdisplayed on the display section 24 as a playback image. When theplayback image has been displayed, shooting conditions or image outputinformation for each region may be displayed depending on the user'sdesignation. In this way it is possible for the user to confirminformation when necessary.

Next, control of shooting conditions using a front curtain electronicshutter of the image sensor 15 will be described using FIG. 3A and FIG.3B. In FIG. 3A, the vertical direction corresponds to up and downdirection of the imaging surface of the image sensor 15, while thehorizontal direction corresponds to time. At time T1, exposurecommences, sequentially, from a pixel line at a lower edge of an imagetowards a pixel line at an upper or edge of the image. If exposurereaches pixel Tc, commencement time is delayed until T2, and if time T2is reached exposure is restarted, sequentially towards a pixel line atan upper edge of the image, similarly to the case of exposure at thelower edge of the image.

Then, at time T3, exposure is terminated, sequentially, from a pixelline at a lower edge of an image towards a pixel line at an upper oredge of the image. As a result of this, as shown in FIG. 3A, at thelower edge of the image exposure time becomes Td (corresponding toshutter speed), while at the upper edge of the image exposure timebecomes Tu (corresponding to shutter speed). In this way, by using thefront curtain electronic shutter, it is possible to change shootingconditions (in this case, shutter speed) at the top and bottom of thescreen, and when shooting one frame it is possible to carry out shootingat a plurality of shooting conditions.

It should be noted that with the example shown in FIG. 3A, the exposurecommencement time point is not changed linearly but is shifted by a timecorresponding to a pixel line Tc, while the exposure completion timepoint is changed linearly. However, this is not limiting, and theexposure commencement time point may be change linearly while theexposure completion is not changed linearly, and both the exposurecommencement time point and the exposure completion time point need notbe changed linearly. Also, with the example shown in FIG. 3A, there isonly a single pixel line Tc for which shutter speed is changed, but thisis not limiting and there may be two or more pixel lines for whichshutter speed is changed. Also, instead of switching shutter speed atthe shutter speed change position (line) Tc, the shutter speed may bechanged gradually before and after Tc. In this case, it is possible toachieve a natural feeling photograph without abrupt changes in imagebrightness before and after the pixel line Tc.

Next, an example where the same effect as in FIG. 3A is achieved byperforming control using applied gain value will be described using FIG.3B. In this embodiment, the gain application section 26 applies gainvalues such as shown in FIG. 3B. However, this is not limiting, and again application circuit may be provided within the image sensor 15,with application of gain values such as shown in FIG. 3B being performedusing this circuit.

In FIG. 3B, the vertical direction corresponds to the up and downdirection of the imaging surface of the image sensor 15, while thehorizontal direction corresponds gain value that is applied. Gain Gd isapplied to pixel values of a pixel line that has been read out from alower edge of the image. A gain Gu is applied to pixel values of a pixelline that has been read out from above a gain value change position(line) Gc.

As a result of applying gain Gu, it is possible to obtain the sameeffect as when exposing with shutter speed Tu, as shown in FIG. 3A, andby applying gain Gd it is possible to obtain the same effect as whenexposing with shutter speed Td. Specifically, when shooting a singleframe shooting can be carried out at a plurality of shooting conditions.

It should be noted that similarly to the case of the electronic shutter,applied gain value may be changed gradually before and after the gainvalue change position (line) Gc. In this case, it is possible to achievea natural feeling photograph without abrupt changes in image brightnessbefore and after the gain value change position (line).

Next, extraction of shooting conditions using shooting conditionprocessing (S15, S31), and display of image generated using theextracted shooting conditions (S23, S41), will be described using FIG.4A and FIG. 4B. It should be noted that both steps of image displayingmay be the same way of displaying, except that image data before imageprocessing has been applied is different.

FIG. 4A shows an image obtained with normal shooting, namely, an imagein the case of shooting with a single shooting condition. With thisexample shown in FIG. 4A, the lower edge of the image is dark. FIG. 4Bis an image that has been generated by making shutter speed and gainvalue different depending on an image region, as was shown in FIG. 3Aand FIG. 3B. Specifically, FIG. 4B is for a case where shutter speed atthe lower part of the image is made longer than the upper part, and gainvalue is increased at the lower part.

With the example shown in FIG. 4A, ISO sensitivity was set to 200, andshutter speed (SS) was set to 1/1000. On the other hand, with theexample shown in FIG. 4B, at the upper part of the screen ISOsensitivity is set to 200 and shutter speed (SS1) is set to 1/1000,while at the lower part of the screen ISO sensitivity is set to 400 andshutter speed (SS2) is set to 1/100. In this way, by dividing the screeninto two regions vertically and displaying shooting conditions forrespective regions, it is possible for the user to ascertain shootingcondition intuitively. Also, a display (LINE) indicating where on theimage shooting conditions (setting values) change may be displayedsuperimposed on the image.

The image shown in FIG. 4B may be displayed using a playback image (S41in FIG. 2), and may be display using a live view image (S23 in FIG. 2).If live view display is used, then it is possible to assume a completedimage beforehand, and in a case where the image is not as intended it ispossible to perform adjustments while looking at the live view image.So, if display of a playback image is used, it is possible to confirmcompletion, and if the image is not as intended it is possible to carryout shooting again after having adjusted shutter speed etc. Also, at thetime of adjustment, besides changing shooting conditions such as shutterspeed, the position of LINE may be displaced up or down.

It should be noted that a case is assumed where a live view image isgenerated under different shooting conditions from actual shooting. Forexample, if, with actual shooting, shooting is carried out with adifferent shutter speed in the regions above and below LINE, then a caseis assumed where live view shows an simulated image that would beacquired with actual shooting with shutter speed being set to a commonvalue in the regions above and below LINE while different gain valuesare set. In this case, since what the user wishes to ascertain with liveview is shooting conditions for actual shooting (shutter speed set inregions above and below LINE), it is preferable to display shootingconditions at the time of actual shooting even if live view shooting hasbeen performed with shooting conditions that are different to those foractual shooting. In this case, therefore, an image sensor carries out anactual shooting operation to shoot image data for a single frame at aplurality of shooting conditions, and image region information and imageassociated information at the time of the actual shooting operation aredisplayed superimposed on the live view image data before actualshooting.

Next, extraction of image output information (S17, S33), and display ofan image that has been generated based on the extracted image outputinformation (S23, S41), will be described using FIG. 5A and FIG. 5B. Itshould be noted that both steps of image displaying may be the same wayof displaying, except that image data before image processing has beenapplied is different.

FIG. 5A shows calculation of shift amounts EV1 and EV2 with respect toproper exposure, as image output information, and is an example in whichhistograms representing brightness distribution are calculated anddisplayed. An output average value of subject brightness for an imageregion that is above the boundary line LINE is calculated, and shift EVwith respect to proper exposure is calculated and displayed as EV1 atthe upper right of the screen. In this case EV becomes 0 at properexposure, EV becomes + if there is overexposure, and EV becomes − ifthere is under exposure.

A shift EV with respect to proper exposure for an image region that isbelow the boundary line LINE is calculated, and displayed as EV2 at thelower right of the screen. With the example shown in FIG. 5A EV2=−2, andrepresents that an exposure value is two steps underexposed from properexposure.

A histogram HIS_1 is shown slightly toward the upper side of the left ofthe screen. This histogram HIS_1 is histogram display of subjectbrightness for an image region above the boundary line LINE. Also, ahistogram HIS_2 is shown at the lower left of the screen. This histogramHIS_2 is histogram display of subject brightness for an image regionbelow the boundary line LINE. The histograms have a curve peak that isshifted to the right as a subject becomes brighter, and a curve peakthat is shifted to the left as a subject becomes darker. The user canintuitively understand the distribution of subject brightness bycomparing histograms of each image region. It should be noted that theboundary line LINE can have its position changed by the user.

Calculation and display of the image output information are carried outas follows. The region information processing section 201 extractsregion information for respective regions, and the image outputinformation processing section 203 calculates pixel output forrespective regions. If there is an EV value representing shift fromproper exposure, then by how many steps an average value of pixel outputis shifted, with respect to proper exposure (according to JPEG255LSB,118LSB is made proper exposure), is calculated. Also, in the events thatdisplay is with histograms, statistical distribution of pixel output forrespective regions is calculated.

Based on calculation results by the region information processingsection 201 and the image output information processing section 203 (EVvalues or histogram graphs), the display image generating section 29generates an image that is superimposed on live view or a playbackimage. As a result the user can ascertain features of images togetherwith exposure of respective regions. If there is live view, exposure ofrespective image region can be adjusted while confirming EV values andhistograms, as shown in FIG. 4B, and it is possible to carry out controlso as to obtain an intended image.

FIG. 5B shows another example of calculating and displaying image outputinformation. With this example results of averaging brightness levelsfor pixels in each horizontal line are made into a graph and displayed.Specifically, brightness levels are averaged for every one line, in apixel line direction (screen horizontal direction), and displayed asshift from proper exposure level. For example, Lp within the screen ofFIG. 5B is a brightness level average value for pixel line P, and isabout +1.4 EV. The graph L is a line that connects brightness levels foreach pixel line. Also, the graph L may be smoothed by taking an averageover a plurality of lines, or subjecting average values of each line tofilter processing.

It should be noted that in FIG. 5B, amount of shift from proper exposureon the horizontal axis is displayed as an EV value, but this is notlimiting and the amount of shift may also be displayed as shutter speed(Tv value). If displayed using shutter speed, it is extremely convenientin a case where shooting conditions for an image region are changedusing a front curtain electronic shutter such as described previously.It is made possible to switch over a display between shutter speeddisplay (Tv value) and EV value display (and brightness value) usingsetting change to be referred to when operating the front curtainelectronic shutter.

In FIG. 5B, in a case where the horizontal axis shows brightness level,average levels for the pixel direction (horizontal direction) arerepresented. Alternatively in a case of showing shutter speed thehorizontal axis may represent Tv values so that with a TV value thatgives proper exposure as a reference, for example, bright portions takea minus value, and dark portions take a positive value, serving as anindex on setting a front curtain electronic shutter. With respect to Tvvalues, if there is representation relating to exposure time, seconds orsetting values inside the camera may be displayed directly. Also, inFIG. 5B, a scale is designated in single step units, but scale width andscale interval may be changed depending on a subject.

Next, another example of image region setting will be described usingFIG. 6A and FIG. 6B. With the examples that were shown in FIG. 4A, FIG.4B, FIG. 5A and FIG. 5B, the boundary line LINE extending in thehorizontal direction was moved up and down by the user using a touchoperation. With the example shown in FIG. 6A, the screen is divided intoa plurality of blocks, the respective blocks are analyzed, and thescreen may be automatically divided into image regions. Shootinginformation and image output information may be extracted for each ofthe divided image regions. With the example shown in FIG. 6A each blockis made up of a plurality of pixels, but an image region may be set foreach single pixel, and shooting information and pixel output informationextracted for each single pixel and stored. However, in this case theinformation amount becomes extremely large. With this embodiment,therefore, as shown in FIG. 6A, information amount is reduced bygrouping into blocks of a few tens of pixels, namely plurality of (x, y)pixels.

With the example shown in FIG. 6B, region division is carried out basedon subject distance. With this example, the image sensor 15 has an imageplane phase difference AF function (capability of detecting distanceinformation at each pixel of the image sensor), and it is possible toascertain subject focus information (subject distance information fromthe camera) for each block. The region information processing section201 carries out division of image regions based on subject distanceinformation for each block.

With the example shown in FIG. 6B, a photograph is taken focused on aflower (regions other than the flower where subject distance isdifferent to that for the flower are shot out of focus, and shooting iscarried out with different shooting conditions (focus conditions) in asingle image). The region information processing section 201 divides aportion of the flower in the foreground into two image regions, being aregion AREA shown by the dashed line, and a background portion (portionsoutside of the region AREA). Then, for the respective regions imageoutput information, subject distance here, is displayed, so for theforeground region AREA 50 cm is displayed as distance 2, and in thebackground portion other than the region AREA ∞ (infinity) is displayedas distance 1.

Also, not only is image output information (subject distance) displayed,but together with image data, image output information is associated andstored for every image region. In this way, it is possible to utilizesubject distance as information on the subject at the time of shootingafter shooting also.

Next, a case of associating and storing shooting conditions and imageoutput information for every image region be described using FIG. 7A andFIG. 7B. FIG. 7A shows a way of dividing regions, in a case whereshooting conditions and image output information are stored, in theshooting condition processing section 202 and image output informationprocessing section 203. Specifically, with the example shown in FIG. 7A,the imaging surface of the image sensor 15 is divided into N (integer)in the X direction and into M (integer) in the Y direction, with aplurality of pixels being included in respective regions. Shootingconditions and pixel output are extracted in each region.

FIG. 7B schematically shows a data structure of an image file.Specifically, FIG. 7B shows one example of a data structure for storingan image region information that the region information processingsection 201 has output by associating with shooting conditions that havebeen extracted by the shooting condition processing section 202, andimage output information that has been extracted by the image outputinformation processing section 203, so as to constitute each of theblocks as shown in FIG. 7A.

An image file IF is associated with metadata MD that is associated withimage data (pixel output data) ID, and stored in the internal memory 22or external memory 23. The metadata MD stores various information suchas camera model name, lens name and shooting date and time etc.

As shown in FIG. 7B, shooting condition and image output information arestored in association with each other in the metadata MD, for everysequence information of a block, which is image region information. In ageneral image file, a single shooting condition is stored as metadataassociated with one frame of image data. However, with this embodiment,even in a case where there are different shooting conditions and imageoutput information for each image region, it is possible to storeinformation for each image region. In this way it is possible toassociate shooting conditions and pixel output of each image region withimage data, and to process and store.

It should be noted that in this embodiment, an example has been shownwhere image region information and image associated information isstored in a metadata storage region of an image file IF, but this is notlimiting. For example, image region information and image associatedinformation may be stored as an associated data file that is differentto the image file IF (for example, stored so as to save the image fileIF and the associated data file within the same folder). Also, forexample, in a case where image associated information is stored forevery one pixel, as a result of making data amount of the imageassociated information large, data amount of the image file IF becomeslarge, and efficiency of data processing is deteriorated (dataprocessing time for playback, copying, and moving of image data isincreased). In this type of case, if it is possible to handle the imagefile IF and the associated data file as separate files then it ispossible to carry out data processing efficiently even if the dataamount of the image region information and the image associatedinformation is large.

In this way, in the first embodiment of the present invention, in a casewhere shooting has been performed with shooting conditions that differin regions within an image of a single frame, image associatedinformation (shooting conditions and image output information) inindividual image regions is associated with image data (S15, S17 and S19in FIG. 2). As a result, it is possible to easily confirm imageassociated information for each image region.

Next, a second embodiment of the present invention will be describedusing FIG. 8 and FIG. 9. With the first embodiment, within an image of asingle frame was divided into a plurality of regions, and even in a casewhere shooting conditions were made different for each region respectiveshooting conditions and image output information was displayed, as wellas being associated with image data and stored in the external memory 23or the like. Conversely, with the second embodiment, images of aplurality of frames are combined to generate an image for a singleframe, with shooting conditions and image output information forrespective frames being displayed, as well as being associated withimage data and stored in the external memory 23 or the like.

The structure of a camera of the second embodiment is similar to that ofthe first embodiment shown in FIG. 1, and so detailed description willbe omitted. Regarding operation of the camera of the second embodiment,the flowchart shown in FIG. 2 is replaced with the flowchart shown inFIG. 8. This flowchart assumes a case where an image combination mode isset in which images for a plurality of frames are taken, and images arecombined. As an image combination mode, there is photo story in whichimage data for a plurality of frame is taken, and combination is carriedout so that each frame is arranged within a single frame, as if to showa single story (refer, for example, to Japanese patent laid-open number2014-068273). Also, besides photo story, there are other modes thatcombine images, such as focus stacking, multiple exposure, HDRcombination etc. It should be noted that in this embodiment image regiondesignation is not carried out for every frame, and display and storageare carried out for conditions of each region of images that have beencombined under various conditions.

If the flowchart shown in FIG. 8 is entered, first live view display iscarried out (S51). Here, similarly to step S1, the aperture 13 is set toa wide-open aperture value and the shutter 14 is placed in an openstate, and a subject image is displayed on the display section 24 basedon image data from the image sensor 15.

If live view display has been carried out, setting of shootingconditions is next carried out (S53). Here, shooting conditions thathave been set by the user are acquired. In the case of photo story,shooting conditions such as shutter speed, aperture value, ISOsensitivity, image processing conditions (for example art filter) etc.are automatically or manually set for each individual frame. In the caseof focus stacking, multiple exposure, HDR combination etc., if aninitial frame has been set, then for second and subsequent framesshooting conditions are set in accordance with predetermined conditions.With focus stacking, a plurality of focus positions (focus lens controlsettings) required to create an image in which the entire image is infocus is set automatically or manually. With HDR combination, shootingfor a plurality of exposure settings (shutter speed, ISO sensitivity oraperture value) so as to be able to create an image having a desireddynamic range is set automatically or manually.

Once setting of shooting conditions has been carried out it is nextdetermined whether or not a release switch is on (S55). Here whether ornot the release switch is on is determined based on detection signalsfor the release button from the input section 25. If the result of thisdetermination is that the release button has not been pressed down,processing returns to step S51.

On the other hand, if the result of determination in step S55 is thatthe release switch is on, an image is read out (S57). If the releaseswitch is on, the system controller 21 controls aperture value of theaperture 13, controls shutter speed of the shutter 14, and controlsimaging by the image sensor 15. Then, if an exposure time determined bythe shutter speed has elapsed, the drive control section 11 performsreadout of image data from the image sensor 15, and outputs to the bus30.

If readout of an image has been performed, next gain is applied to theimage data (S59). Here the gain application section 26 applies gained tothe image data. It should be noted that in a case where it is possibleto apply gain when reading out image data from the image sensor 15, gainmay also be applied when reading out from the image sensor 15 in stepS57.

If gain application has been carried out, next development processing iscarried out (S61). Here the development processing section 28 subjectsimage data to processing.

Once development processing has been carried out, next shootingcondition processing is carried out (S63). Here the shooting conditionprocessing section 202 extracts shooting conditions for at the time ofshooting (shutter speed, gain etc.) from the system controller 21. Also,image processing settings that have been set in the image data areextracted from the development processing section 28. When extractingshooting conditions, the conditions are stored by associating withinformation indicating what frame in a sequence has been taken Asshooting conditions, for example, at least one of gain, shutter speed,focal length (zoom position), whether or not a flash is being used,aperture value, exposure control information (under/correct/over)shooting date (date and time when the release button was pressed down)etc. is extracted.

Once shooting condition processing has been carried out, next imageoutput information processing is carried out (S65). Here, the imageoutput information processing section 203 calculates image outputinformation from image data. As image output information, for example,at least one of histogram information, exposure (shift from properexposure, EV value), noise value, subject distance etc. is extracted.

Once image output information processing has been carried out, nextimage data is stored (S67). Here, image data of a single frame that wasread out in step S57 is stored in the internal memory 22 or the externalmemory 23. For example, in a case where continuity of shooting isrequired, such as for focus stacking etc., the internal memory 22 thatis capable of reading and writing data is preferable. Also, in a casewhere rapid shooting is not required, such as in photo story, storage iscarried out to the external memory 23. Also, when storing image data ofa single frame, shooting conditions that have been extracted by theshooting condition processing of step S63 and image output informationthat has been extracted by the image output information processing ofstep S65 are associated with image data, and an image file is generatedand stored.

Once image data has been stored it is next determined whether or not arequired number of frames has been reached (S69). In an imagecombination mode, there are cases where a required number of frames forcombination has been determined. In the event that a number of frameshas been determined, in this step it is determined whether or notshooting has been completed for the required number of frames. If arequired number of frames has not been determined, then it is madepossible for the user to continue shooting until shooting is complete,and in this step it is determined whether or not shooting is complete.If the result of determination in this step is that the required numberof frames has not been reached processing returns to step S51, shootingconditions for the next frame are set, and shooting is carried out. Itshould be noted that in a case where, once shooting is commenced, apredetermined number of frames are captured continuously, such as withfocus stacking, the determination in step S55 may be skipped andprocessing advanced to step S57.

If the result of determination in step S69 is that the required numberof frames has been reached, next image combination is carried out (S71).If shooting is complete, images of a plurality of frames that have beentaken are combined by the image combination section 27, and a combinedimage is stored in the internal memory 22. For example, if there is aphoto story, change of image size is carried out on the size of eachframe that has been subjected to frame division, and processing to fitimages within a single frame is carried out (referred to FIG. 9). Also,if there is HDR combination, combination is performed by carrying outweighting of outputs of the same pixel addresses for a plurality offrames that have been taken with changed exposure, in accordance withsubject brightness. If there is focus stacking, shooting of a pluralityof frames is carried out with focus position shifted, and images arecombined so as to make a shift in focus for each position of the imagesmall.

Once image combination has been carried out, next region informationprocessing is carried out (S73). With the first embodiment a singleframe was divided into image regions, but with the second embodiment aplurality of frames are subjected to image combination, and the framescorrespond to respective image regions. In this step shooting conditionsand image output information are extracted for each frame, andassociated with image data of each frame.

For example, in a case where photo story has been set as a combinationmode, the region information processing section 201 extracts addressinformation of each frame (for example, in a case where a frame issquare, coordinates of the top) as image region information, theshooting condition processing section 202 and the image outputinformation processing section 203 extract shooting conditions and imageoutput information of image data that has been allocated to respectiveframes, and these items of information are associated with each frame.Also, in a case where HDR combination has been set and, for example,images that have been taken with exposure at −1EV, 0EV, and +1EV arecombined, in the combined image, image data about which exposure hasbeen weighted the most and combined may be extracted as a shootingcondition for each block such as shown in FIG. 7A. In a case where focusstacking mode has been set, each block of the image may have at whatfocus position an image that has been weighted the most was taken andcombined extracted as a shooting condition (shooting distance) for eachblock.

Once region information processing has been carried out, next image datais stored (S75). Here, image data that was combined in step S71, andmetadata that was associated in steps S73 and S75, are stored in theexternal memory 23 as an image file.

If storage of image data has been carried out, next generation of adisplay image is carried out (S77). Here, the display image generatingsection 29 generates an image in which the shooting conditions and imageoutput information of each frame are superimposed on the image data.

Once a display image has been generated, next display of a playbackimage is carried out (S79). Here, a display image that was generated instep S77 is displayed on the display section 24 as a playback image.Once the playback image has been displayed this flow is terminated.

In this way, in this sequence a plurality of images for combination aretaken (S57), and shooting conditions and image output information ofeach frame are extracted and stored (S63-S67). Then, if an image for asingle frame has been generated by carrying out image combination (S71),shooting conditions and image output information for each image region(specifically each block shown in FIG. 7A) of this combined image areextracted (S73). The shooting conditions and image output informationthat have been extracted are stored, and made displayable (S75-S79).This means that with this embodiment, even when combining from aplurality of frames, it is possible to easily confirm image regions andimage associated information.

It should be noted that with this sequence combination processing isexecuted after completion of shooting for a plurality of frames, butthis is not limiting, and it is also possible to sequentially combineimages and process the images at an image information processing sectionwhile carrying out shooting. By carrying out this type of processing itis possible, while confirming image information (shooting conditions andimage output information of each frame) during combination, to carry outshooting while adjusting shooting conditions for the frames. Also, withshooting conditions other than those described above, as long asinformation is information that can be obtained by a general imagingapparatus, the information can be similarly processed. With image outputconditions other than those described above also, if it is numericalinformation obtained with a general image processing apparatus, similarprocessing is possible.

Next, display of image associated information (shooting conditions andimage output information) for a case where photo story has been set asan image combination mode will be described using FIG. 9. With theexample shown in FIG. 9, three images are taken, and a combined imagefor a single frame is generated by respectively placing these threeimages at the left side, upper right, and lower right of the screen. ISOsensitivity, shutter speed (SS), aperture value (F value), focal length,and type of art filter that has been used are displayed superimposed onthe respective images as shooting conditions and image outputinformation. This means that even in a case where a combined image of asingle frame has been generated from a plurality of images, it ispossible to easily confirm shooting conditions and image outputinformation of individual images.

As has been described above, with the second embodiment, even if animage for a single frame has been generated by combining a plurality ofimages, image associated information of individual images (shootingconditions and image output information) are correlated with image data(S63, S65 and S73 in FIG. 8). As a result, it is possible to easilyconfirm individual image associated information for a plurality ofimages.

As has been described above, with each of the embodiments of the presentinvention, a storage section (for example, the internal memory 22 inFIG. 1) that stores image data, an image associated informationprocessing section (for example, the image associated informationprocessing section 200) that, for image data of a single frame that hasbeen taken under a plurality of shooting conditions, associates andprocesses image region information, relating to image regions whereshooting is carried out under different shooting conditions, and imageassociated information of the image regions, are provided. Morespecifically, image data of a single frame is either image data shotunder different shooting conditions for each image region of an image ofa single frame (refer to the first embodiment), or image data formed bycombining a plurality of images taken under different shootingconditions (refer to the second embodiment). In this way, with each ofthe embodiments, since image associated information for each imageregion is associated with image data by the image associated informationprocessing section 200, it is possible to easily confirm imageassociated information for each image region.

Also, in each of the embodiments of the present invention, a displaysection (for example, the display section 24) is provided that displaysimage region information and image associated information superimposedon image data. The image region information is information representingregions within an image that have been divided by the boundary line LINEin FIG. 4B and FIG. 5A, for example, and is information on regionsrepresented by blocks as shown in FIG. 6A and FIG. 7A. In the lattercase, it is possible to display respective blocks using (x, y)coordinates. In each of the embodiments, since image associatedinformation is displayed by the display section superimposed on animage, it is possible to grasp the image associated informationintuitively.

Also, with each of the embodiments of the present invention, image dataof a single frame that has been taken under a plurality of shootingconditions is image data that has been formed by an image sensor (forexample, the image sensor 15). As a result, when shooting with theimaging apparatus (also including live view) it is possible to easilyconfirm image associated information, as described above.

Also, with each of the embodiments of the present invention, an imagesensor is capable of varying shutter speed for every pixel or for everyline, and shutter speed is handled as image associated information. Aswas described above, in a case where the image sensor is provided withan electronic shutter, it is possible to vary shutter speed for everypixel or for every line. As a result it is possible to vary shutterspeed for every image region.

Also, with each of the embodiments of the present invention, imageassociated information is at least one of shutter speed, ISOsensitivity, focal length and exposure control value (for example, EVvalue), which are shooting conditions for each image region. Also, theimage associated information is values calculated using pixel outputincluded in every image region, and is at least one of statistical valueof pixel output (for example, histogram, EV value), exposure shiftamount (for example EV value), and subject distance. In this way, witheach embodiment of the present invention, it is possible to use variousinformation as the image associated information, and it is possible toconfirm information for every image region.

Also, in each of the embodiments of the present invention a gainapplication section is provided that applies gain to image output forevery pixel or for every line, and gain values are treated as imageassociated information. This means that it is possible to vary gain forevery image region.

Also, with each embodiment of the present invention, an imagecombination section (for example the image combination section 27 inFIG. 1) is provided that combines image data of a plurality of frames,and image data of a single frame, that has been taken at a plurality ofshooting conditions, is image data that has been combined by the imagecombination section. This means that it is possible to easily confirmindividual shooting conditions, for an image that has been taken under aplurality of shooting conditions.

Also, with each embodiment of the present invention, there is providedan image data generating section (for example, the image data generatingsection 204 of FIG. 1) that stores image region information that hasbeen associated with image data by the image associated informationprocessing section, and image associated information of image regions,as metadata of the image data (for example, FIG. 7B). Since imageassociated information is stored as metadata of the image data, it iseasy to associate and display the information.

Also, with each embodiment of the present invention, a shootingcondition input section (for example, the input section 25 of FIG. 1) isprovided that receives inputs of shooting conditions from thephotographer, and shooting conditions for each of a plurality of imageregions are set in accordance with signals from the shooting conditioninput section. As a result it is possible for the user to manually setshooting conditions.

Also, with each of the embodiments of the present invention, shooting iscarried out with different shooting conditions and image data of asingle frame is stored (for example, S25 in FIG. 2 and S71 in FIG. 8),and for image data of a single frame image associated information isextracted (for example, S31 and S33 in FIG. 2 and S63 and S65 in FIG.8), and association of image data for a single frame and imageassociated information for every image region is carried out (forexample, S35 in FIG. 2 and S73 in FIG. 8). As a result, it is possibleto easily confirm image associated information for each image region.

Also, with each of the embodiments of the present invention, image datafor a single frame is either, image data taken with different shootingconditions for each image region of a single shot (for example, S3-S23in FIG. 2), or a combined image formed by combining image data acquiredby shooting a plurality of times (S51-S69 in FIG. 8). This means that itis possible to easily confirm image associated information for eachimage region, even in a case where a plurality of shooting conditionshave been set using various shooting methods.

It should be noted that in each of the embodiments of the presentinvention the image processing section 20 has been constructedseparately from the system controller 21, but all of or some sections ofthese two components may be constructed together. For example, the imageassociated information processing section 200 may be implemented by aCPU within the system controller 21.

Also, with each of the embodiments of the present invention, some or allsections of the image processing section 20 and the system controller 21may have a hardware structure such as gate circuits generated based on aprogramming language that is described using Verilog, and also ahardware structure that utilizes software such as a DSP (digital signalprocessor) may be used. Suitable combinations of these approaches mayalso be used.

Also, with each of the embodiments of the present invention, aninstrument for taking pictures has been described using a digitalcamera, but as a camera it is also possible to use a digital single lensreflex camera or a compact digital camera, or a camera for movie usesuch as a video camera, and further to have a camera that isincorporated into a mobile phone, a smart phone, a mobile informationterminal, personal computer (PC), tablet type computer, gain consoleetc., or a camera for a scientific instrument such as a microscope, acamera for mounting on a vehicle, a surveillance camera etc. In anyevent, the present invention may be applied to any apparatus that cancarry out shooting with varied shooting conditions.

It is possible, for example, to apply the present invention even with animage processing apparatus (such as a personal computer (PC)) where,with image data of a plurality of frames that have already been taken,image data having shooting conditions and pixel output information etc.stored as metadata are combined. Also, even if an imaging section is notprovided, it is possible to apply this application as long as it is to adevice that carries out processing to associate shooting conditions andimage output information, for every image region (every pixel) of imagedata for a single frame resulting from combination of a plurality ofimages, with image data.

It is also possible to similarly apply the present invention as long asit is to a device provided with an image processing apparatus, such as asmart phone, microscope, or endoscope. For example, with a microscope orthe like, in a case where images for a plurality of frames have beentaken with varying light source wavelengths, when treating an image of aplurality of frames taken with varying wavelength as image data for asingle frame, information on wavelength at the time of shooting eachframe may be stored as measurement information.

Also, among the technology that has been described in thisspecification, with respect to control that has been described mainlyusing flowcharts, there are many instances where setting is possibleusing programs, and such programs may be held in a storage medium orstorage section. The manner of storing the programs in the storagemedium or storage section may be to store at the time of manufacture, orby using a distributed storage medium, or they be downloaded via theInternet.

Also, regarding the operation flow in the patent claims, thespecification and the drawings, for the sake of convenience descriptionhas been given using words representing sequence, such as “first” and“next”, but at places where it is not particularly described, this doesnot mean that implementation must be in this order.

As understood by those having ordinary skill in the art, as used in thisapplication, ‘section,’ ‘unit,’ ‘component,’ ‘element,’ ‘module,’‘device,’ ‘member,’ ‘mechanism,’ ‘apparatus,’ ‘machine,’ or ‘system’ maybe implemented as circuitry, such as integrated circuits, applicationspecific circuits (“ASICs”), field programmable logic arrays (“FPLAs”),etc., and/or software implemented on a processor, such as amicroprocessor.

The present invention is not limited to these embodiments, andstructural elements may be modified in actual implementation within thescope of the gist of the embodiments. It is also possible form variousinventions by suitably combining the plurality structural elementsdisclosed in the above described embodiments. For example, it ispossible to omit some of the structural elements shown in theembodiments. It is also possible to suitably combine structural elementsfrom different embodiments.

What is claimed is:
 1. An image processing apparatus comprising: amemory that stores first image data, and a processor that includes animage associated information processing section, wherein the imageassociated information processing section, for the image data of asingle frame that has been taken at a plurality of shooting conditions,within the first image data that has been stored in the memory, acquiresimage region information, relating to image regions in which shooting iscarried out at different shooting conditions, and image associatedinformation of the image regions, associates the image regioninformation and the image associated information and subjects the firstimage data to image processing, and generates second image data.
 2. Theimage processing apparatus of claim 1, further comprising: a displaythat displays the image region information and the image associatedinformation superimposed on the first image data.
 3. The imageprocessing apparatus of claim 1, further comprising: an image sensorthat generates image data from a subject image, wherein image data forone frame that has been taken at the plurality of shooting conditions isimage data that has been formed by the image sensor.
 4. The imageprocessing apparatus of claim 3, wherein the image sensor is capable ofvarying shutter speed for every pixel or for every line, and shutterspeed is treated as the image associated information.
 5. The imageprocessing apparatus of claim 1, wherein the image associatedinformation is at least one of shutter speed, ISO sensitivity, focallength and exposure control value (EV value), which are the shootingconditions for each image region.
 6. The image processing apparatus ofclaim 1, wherein the image associated information is at least one of astatistical value for pixel output, exposure offset amount or subjectdistance, which are values that are calculated using pixel outputcontained in each of the image regions.
 7. The image processingapparatus of claim 1 wherein, the processor further comprises: a gainapplication section that applies gain to image output for every pixel orevery line, and wherein the image associated information processingsection treats gain values as the image associated information.
 8. Theimage processing apparatus of claim 1 wherein, the processor furthercomprises: an image combination section that combines image data of aplurality of frames, and wherein image data for one frame that has beentaken at the plurality of shooting conditions is image data that hasbeen combined by the image combination section.
 9. The image processingapparatus of claim 1 wherein, the processor further comprises: an imagedata generating section that stores image region information that hasbeen associated with image data by the image associated informationprocessing section, and image associated information for the imageregions, as metadata of the image data.
 10. The image processingapparatus of claim 1, further comprising: a shooting condition inputsection that receives input of shooting conditions from a photographer,wherein the image associated information processing section setsshooting conditions for each of the plurality of image regions inaccordance with signals from the shooting condition input section. 11.The image processing apparatus of claim 3, wherein: the image sensorcarries out an actual shooting operation to shoot image data for asingle frame at a plurality of shooting conditions, and before actualshooting the image region information and the image associatedinformation for at the time of the actual shooting operation aresuperimposed on the live view image data to be displayed.
 12. An imageprocessing method comprising: carrying out shooting with differentshooting conditions, and storing image data of a single frame,extracting image associated information for the image data of a singleframe, and carrying out association of the image data of a single frameand the image associated information for each image region.
 13. Theimage processing method of claim 12, wherein the image data for a singleframe has different shooting conditions for each of the image regions ofa single shot.
 14. The image processing method of claim 12, wherein theimage data for a single frame is generated by image combination of imagedata that has been acquired by carrying out shooting a plurality oftimes.